Basin Entropy in Boolean Network Ensembles 
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The information processing capacity of a complex dynamical system is reflected in the partitioning 
of its state space into disjoint basins of attraction, with state trajectories in each basin flowing towards 
their corresponding attractor We introduce a novel network parameter, the basin entropy, as a measure 
of the complexity of information that such a system is capable of storing. By studying ensembles of 
random Boolean networks, we find that the basin entropy scales with system size only in critical regimes, 
suggesting that the informationally optimal partition of the state space is achieved when the system is 
operating at the critical boundary between the ordered and disordered phases. 



Introduction Complex dynamical systems generically 
possess state spaces that are partitioned into disjoint basins 
of attraction. Within each basin, state trajectories flow to a 
single attractor, which, in general, may be a steady-state 
fixed point, an oscillation - including limit cycles, or a 
strange (chaotic) attractor. Such classification of states via 
attractors and their basins of attraction constitutes the in- 
formation processing ability of a complex system and rep- 
resents a type of associative memory: all states in a basin 
are in the same class in that they are all associated with the 
same attractor. The complexity of information that such a 
system is capable of processing depends on the manner in 
which its state space is partitioned, which in turn is largely 
determined by the dynamical regime of the system. By 
using a novel network parameter, the basin entropy, and 
the model class of random Boolean networks, for which 
there exist well-defined and extensively studied notions of 
ordered, critical, and chaotic dynamics [1],[2],|3],|4], we 
show here that the informationally optimal partition of the 
state space is achieved when the system is operating at 
the critical boundary between the ordered and disordered 
regimes. 

Random Boolean networks (RBNs) are commonly stud- 
ied as generic models of dynamically interacting entities, 
such as gene regulatory networks. A RBN, originally intro- 
duced by Kauffman \5] (also called Kauffman networks), 
consists of n nodes, each of which can have two possible 
values (0 and 1). Each node receives input from k ran- 
domly chosen nodes that determine its value at the next 
time step via a randomly chosen Boolean function assigned 
to that node. The output of the function is chosen to be 
1 with probability p, known as the bias |6]. A state of 
the network is the collective activity of the nodes. All 
nodes are updated synchronously and the network transi- 
tions from one state to another, thus tracing out a state tra- 
jectory that eventually flows into a series of periodically 
recurring states called an attractor. The transient states that 
flow into an attractor constitute the basin of attraction for 
that attractor. 

In the limit of large n, RBNs exhibit a phase transi- 
tion between a dynamically ordered and chaotic regime. 
Depending on the parameters k and p, small perturba- 



tions die out over time in the ordered regime and increase 
exponentially in the chaotic regime (see e.g., R]). Net- 
works that operate at the boundary between the ordered 
and the chaotic phase have been of particular interest as 
models for gene regulatory networks, as they exhibit com- 
plex dynamics combined with stability under perturbations 
iItIi.iIsIi.iIqIi. iIToIi . The average length of attractors have been 
extensively studied numerically and analytically for a wide 
range of Kauffman networks. In the chaotic phase, the av- 
erage length of attractors increases exponentially with sys- 
tem size [6]. In the highly chaotic case, where the state 
space of a RBN can be approximated by a random map, 
the expected number of attractors increases linearily with 
system size [27]. In the ordered phase, where the fraction 
of nodes that freeze to a constant value approaches one, 
the average number and length of attractors is bounded 
H. lfisll . In contrast to former assumptions, it has been 
recently shown that the average number and length of at- 
tractors increases superpolynomially in critical Kauffman 
networks |28n. 



In this Letter, we introduce a new network parameter, 
the basin entropy /i of a Boolean network, and show that 
this quantity increases with system size in critical ensem- 
bles, whereas it reaches a fixed value in the ordered and 
in the highly chaotic phase. The partition of the state space 
into basins of attraction induces a probability mass function 
over the state space, with the weight of each basin defined 
by its size relative to the other basins. The basin entropy 
(hereafter, simply entropy) of a network is then calculated 
from this probability distribution. The partition that a given 
network imposes on its state space is not unique and thus, 
neither is its entropy. Certain network instances in the or- 
dered phase may have the same entropy as some networks 
in the chaotic phase, and thus it becomes necessary to study 
the average entropies of network ensembles. 

In the informational sense, the entropy of a Boolean net- 
work is a measure of the uncertainty about its dynamical 
behavior. In a network with higher entropy, more infor- 
mation is needed to describe its future behavior and its ef- 
fective complexity is higher |[l4|. Thus, if the network is 
viewed as a classifier, then the entropy is a measure for the 
complexity of this classification process. Herein, we show 
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that an increase in such classification complexity as a func- 
tion of system size occurs only when the system is poised 
at the boundary between an ordered and disordered phase. 

A Boolean Network B = (N, F) is defined by the set 
N of its nodes, N = {1, n}, and the set F of their cor- 
responding Boolean updating functions F = {/i, /„}, 

with fi : {0, 1}*^ {0, 1}. The value x.i of node i at 
time t + 1 is determined by the values of its k controlling 
elements as x,{t + 1) = (t), Xi(fc) (t)). 

The sensitivity s of a network, defined as the average 
sensitivity of the Boolean functions used in the network 
1 150, is an order parameter that specifies the average num- 
ber of nodes that are affected by a perturbation of a random 
node. Thus, an average sensitivity of s = 1 indicates that a 
perturbation of a random node is on average propagated to 
one other node. This defines the point of phase transition 
between the ordered regime (s < 1), where perturbations 
die out over time, and the chaotic regime (s > 1), where 
even small perturbations increase over time [4]. In classical 
Kauffman networks, the relationship between the network 
parameters k and p and the average sensitivity s is 1150 : 
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FIG. 1: Average entropy in ensembles B(n, fe,p = 1/2) with 
n = 10, .., 20 and k — 1, 10. The average entropy grows with 
n for the critical ensembles B(n, fc — 2,p = 1/2) and approaches 
a finite value in the chaotic regime independent of n. 



s = 2kp{l-p). (1) 

The logarithm of this parameter may also be interpreted as 
the Lyapunov exponent, A = logs f^. In the following, 
we study ensembles B(n, k,p) of Boolean networks with 
n nodes parameterized by k and p. For p = | all net- 
works of the ensemble have equal probability. In the gen- 
eral case, where certain Boolean functions might be chosen 
with different probabilities, we indicate by Vi the probabil- 
ity of a certain network instance i in the ensemble. We will 
also find it convenient to refer to the ensemble of networks 
B(n, s) with average sensitivity s. 

Average basin entropy Any Boolean network B G 
B(n,s) partitions its state space into attractors and cor- 
responding basin states. The weight Wp of an attractor p 
is the length of its attractor plus the number of basin states 
draining into that attractor, normalized by the size of the 
state space (2"), so that Wp = 1. The basin entropy h 
of a network B is defined as: 

h{B) = —''^^^WphiWp. (2) 

p 

The average entropy (/i) of an ensemble B(n, s) of net- 
works is defined as: 

{h)\B{n,s)] = - ^ v.^^WpAnWp^, (3) 

ieB(n,s) pi 

where Vi is the probability of a network instance in the 
ensemble. To determine the state space partition of a 
network exactly, one has to link each state to its attrac- 
tor. We performed an exhaustive computation of the state 



space partition to estimate the average entropy of the en- 
sembles 'B{n,k,p = 1/2), with n = 10, . . . , 20 and 
k = 1, . . . , 10. As shown in Fig. ([T]), the average entropies 
of the critical ensembles k = 2 grow with n whereas the 
average entropies of the chaotic ensembles k > 2 approach 
a finite value of approximately 0.61, independent of n. Be- 
fore considering the scaling behavior of the average en- 
tropy in critical network ensembles B(n, s = 1), we will 
discuss the average entropy in the chaotic regime in the 
limit of large n. 

When /c reaches n, the ensemble B(n, A; = n,p = 1/2) 
can be identified with the random map model, which is a 
simple disordered system with deterministic dynamics. For 
each state in the state space, another state (not necessarily a 
different one) is randomly chosen as its successor. Thus, a 
random map can be regarded as an unbiased Boolean net- 
work where each node depends on all n variables. The 
average sensitivity of the random map is thus s = n/2 and 
its average entropy can be expressed as: 

2" 

(h) [B(n, s = n/2)] = - ^ g^w, Inw, (4) 

where the sum is taken over all possible weights Wj G 
{1/2", 2/2" - 1..., 1}, multiplied by the normalized fre- 
quency Qj of their occurrence in the ensemble B(n, s = 
n/2). 

As n cxD, we may replace the sum by an integral, 
where g{w) dw indicates the average number of attractors 
with a weight between w and w + dw: 
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lim {h) [B(n, s = n/2)] = — g{w)wlog{w)dw. 

(5) 

Derrida derived the following analytical expression for 
the distribution of g lfl6ll : g{w) = ^w^^{l — 
w)~i. Substituting this expression into Eq. dSj 
yields lim„_oo (h) [B(n, s = n/2)] = 2 (1 - In 2) = 
0.613 71, which is the average entropy of the random map 
in the limit of large n. It is remarkable that ensembles of 
small network sizes already approach this value soon af- 
ter entering the chaotic regime (see horizontal black line in 
Fig.lB. 

Before discussing the scaling behavior in critical ensem- 
bles, let us briefly recapitulate the concept of irrelevant and 
relevant nodes in a network L2J,L6J and their meaning in the 
context of entropy. A node that is updated by a constant 
Boolean function is a frozen node. All nodes, that even- 
tually take the same constant value on every attractor are 
called clamped nodes lll3ll . These nodes build the frozen 
core of the network. A relevant node is one that eventually 
influences its own state. The number and the length of the 
attractors in a network can be determined from the set of 
relevant nodes. 

Nonfrozen nodes that do not influence their own state 
are irrelevant for the attractor dynamics and are therefore 
called non-frozen irrelevant nodes. As soon as the network 
reaches an attractor, the future behavior of such a non- 
frozen irrelevant node can be determined as a function of 
only the relevant nodes. When we add a nonfrozen irrele- 
vant node to a network, the number of states flowing into 
each of the attractors doubles. Thus, the attractor weights 
do not change by adding or removing nonfrozen irrelevant 
nodes. Further, the addition of a frozen node does not, on 
average, change the attractor weight distribution. Thus, the 
average entropy of an ensemble only depends on the orga- 
nization of its relevant nodes. A set including all relevant 
nodes can be obtained by iteratively removing nodes that 
freeze and become part of the frozen core and non-frozen 
nodes that only influence irrelevant nodes. The remaining 
set is also referred to as the computational core of a net- 
work and only exists in the critical and chaotic regime with 
high probability [31],|30]. The organisation of the relevant 
components is crucial for the understanding of the entropy 
h, as the entropy of the entire network can be calculated 
from the entropy of the independent non-connected rele- 
vant components j: 




FIG. 2: Average entropy {h) in critical k = 1 networks. The 
symbols '+' indicate the results from simulations, {h) approaches 

y^ln 2 - Er=o from atiove. 



tors, and thus the average entropy, is bounded for large sys- 
tems. These results indicate that increasing the system size 
does not, on average, increase the entropy of networks in 
the ordered or chaotic regimes, implying that complexity 
of classification cannot be increased in these regimes. 

Critical ensembles We first treat the special case of a 
critical network with connectivity k = 1. In these net- 
works, all nodes are updated by either the Boolean 'copy' 
or 'invert' function. The topology of such networks con- 
sists of loops and trees rooted in loops. Only nodes on 
loops are relevant. Frozen nodes do not exist in these net- 
works. Consequently, attractors of the same length have 
the same weight. The entropy of a critical k = 1 network 
is the sum of the entropies of its loops. In the limit of large 

n, the number of relevant nodes scales as ~ (^) ^ (a 
critical k = 1 network is a random mapping digraph and 
the number of relevant nodes is equivalent to the number of 
vertices on cycles; see, e.g. ifisll and lfl9h . chapter 14). The 
probabilities of having n; loops of size I are independent 
and Poisson distributed with mean X = ifllll . If we 
approximate the entropy of a loop of size I hy hi ~ / In 2 
and take the sum over the expected number of such loops 
for I < rir, we get the following scaling behavior for the 
entropy in critical k = 1 networks ll32ll : 



h = Y,h, (6) 

j 

For networks in the ordered regime, the number of rele- 
vant nodes approaches a finite limit for large system sizes 
il^l and the proportion of frozen nodes in the network ap- 
proaches one ||2i]. Therefore, the mean number of attrac- 



rir , 1 

hi , /7rn\ 2 
ih) ^^^=ln2.{-) . (7) 

1=1 

In networks with a connectivity higher than one, relevant 
nodes may depend on more than just one relevant node. 
In addition to simple loops, such networks may contain 
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FIG. 3: Average entropy of critical k = 2 and fc = 3 ensembles, 
estimated from random sampling of the state space. The data 
were fitted to the scaling behavior of relevant nodes. 



complex relevant components with crosslinks. The scal- 
ing behavior of relevant nodes in a general class of critical 
random Boolean networks and structural properties of the 
complex relevant components were recently discussed in 
an extraordinary series of publications by Drossel, Kauf- 
man and Mihaljev |[il],||25!],ll23l- The key result of their 
work is that the number of relevant nodes scales as 
~ n3 with the system size in all critical ensembles 



n, 

with k > 1. Furthermore, the proportion of relevant nodes 
that depend on more than one relevant input approaches 
zero with growing n. From Theorem 1.3 by Cooper and 
Frieze in 1 23] we find that the expected number of nodes on 
simple loops increases if the probability of relevant nodes 
depending on one relevant input increases. As we already 
discussed the scaling behavior of the average entropy of 
simple loops, it follows that the average entropy in all criti- 
cal ensembles increases with system size. Fig. [3]shows the 
scaling behavior of the estimated average entropy of the 
critical ensembles k = 2 and = 3 forn = 20, . . . , 500 
(the weight distribution of every network was estimated by 
linking over 2000 random states to their attractors and the 
entropy was then averaged over more than 2000 random 
networks from each ensemble). For both ensembles, we 
fitted the data to an expected number of ~ rele- 
vant nodes. In both cases, the average entropy grows with 
the expected number of relevant nodes. In most cases a 
network will only contain one giant complex component. 
Relevant nodes in this component that depend on more than 
one input will influence the entropy of this component and 
may also explain the different average entropy in different 
critical ensembles. 

Conclusions We have defined the average basin en- 
tropy of an ensemble of Boolean networks and examined 



this parameter for networks in the ordered, critical and 
chaotic regimes. We showed that the average entropy in- 
creases with system size in several critical network ensem- 
bles, but not in the ordered or chaotic regimes, suggesting 
that this is a general property of critical networks under 
synchronous updating schemes. 

Our findings are particularly relevant to the study of bi- 
ological networks, as the information processing in liv- 
ing systems is based on massively parallel dynamics in 
complex molecular networks that underly ontogeny, im- 
mune responses and cognitive behaviour. Kauffman hy- 
pothesized that attractors of genetic regulatory networks 
can be viewed as cell types [71]. If so, then cell types, 
like classes in parallel processing networks, drain basins 
of attraction. Recent experimental evidence supports this 
hypothesis i22ll . 

The cell is also able to respond to changes in its environ- 
ment by changing its own behaviour. Examples of such 
decisions include initiation of the cell division (cell cy- 
cle), execution of specific functions (differentiation), pro- 
grammed cell death (apoptosis), and other cellular func- 
tional states. Such decision-making motivates the view that 
a living system "classifies" its environment according to 
its steady states. When viewed in terms of information 
processing, a cell "reads" the information in its environ- 
ment, propagates that information through its intracellu- 
lar networks of interacting biomolecules and responds by 
exhibiting a corresponding steady-state. The information in 
the steady-state reflects the information in the environment. 
If maximal classification complexity confers a selective ad- 
vantage, our results lend also support to the long-standing 
hypothesis that living systems are critical. 

The classical Kauffman network ensembles studied 
herein have characteristics that do not reflect certain as- 
pects of biological networks. First, the synchronous up- 
dating scheme does not reflect the fact that elements in 
biomolecular networks exhibit different time scales. Sec- 
ond, the organization of biological networks is highly mod- 
ular, containing many loosely coupled modules of similar 
sizes. In contrast, most relevant nodes in random Boolean 
networks are, with high probability, organized into one gi- 
ant component. Since the total entropy is the sum of the en- 
tropies of the components, a growing number of such regu- 
latory modules will certainly increase entropy, independent 
of the updating scheme. Thus, understanding the entropy 
of complex relevant components under different updating 
rules is an important topic for future studies. 
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